Asymmetric Scheduling and Load Balancing for Real-Time on Linux SMP

The ARTiS system, a real-time extension of the GNU/Linux scheduler dedicated to SMP (Symmetric Multi-Processors) systems is proposed. ARTiS exploits the SMP architecture to guarantee the preemption of a processor when the system has to schedule a real-time task. The basic idea of ARTiS is to assign a selected set of processors to realtime operations. A migration mechanism of non-preemptible tasks insures a latency level on these real-time processors. Furthermore, specific load-balancing strategies allows ARTiS to benefit from the full power of the SMP systems: the real-time reservation, while guaranteed, is not exclusive and does not imply a waste of resources. ARTiS have been implemented as a modification of the Linux scheduler. This paper details the evaluation of the performance we conduct on this implementation. The level of observed latency shows significant improvements when compared to the standard Linux scheduler. 1 Real-Time Scheduling on Linux SMP Nowadays, several application domains require hard real-time support of the operating system: the application contains tasks that expect to communicate with dedicated hardware in a time constrained protocol, for example to insure real-time acquisition. Many of those same real-time applications require large amount of computational power. A well known and effective solution to face this requirement is the usage of SMP (Symmetric Multi-Processors). Furthermore, to ensure the durability of application developments, one may prefer to target an operating system that conforms to standards. Despite the definition of a standard POSIX interface for real-time applications [3], each vendor of real-time operating system comes with a dedicated API. From our point of view, this segmented market results from the lack of a major player in the real-time community. To face this situation, we believe in the definition of an Open Source operating system that may federate the real-time community. An real-time extension of the well established GNU/Linux operating system is an attractive proposition. Additionally, it will allow the cohabitation of real-time and general purpose tasks in the system. ⋆ This work is partially supported by the ITEA project 01010, HYADES At a first glance, real-time properties relies on the proposition of a dedicated scheduling policy. Three kinds of modification to the Linux scheduler have been proposed. One consists in running the real-time tasks in a special designed kernel running in parallel, this is what does RTAI [2]. The drawback is that the programming model and configuration methods are different from the usual one: Linux tasks are not real-time tasks and real-time activities can not benefit of the Linux services. The second way taken is to directly modify the Linux scheduler to minimize the path from a blocking context to the re-scheduling of a real-time task. This is, for instance, the base of the work that Ingo Molnar currently carries on. The patch, called “preempt-rt”, focuses on hard real-time latencies (which is new, as all the patches before only focused on soft real-time constraints). The objective is to allow every part of the kernel to be preempted, including critical sections and interrupt handlers. The drawback is the degradation of performance for some system calls as well as the high technical difficulty to write and verify those modifications. The third approach relies on the shielded processors or Asymmetric MultiProcessing principle (AMP). On a multi-processor machine, the processors are specialized to real-time or not. Concurrent Computer Corporation RedHawk Linux variant [1] and SGI REACT IRIX variant [8] follow this principle. However, since only RT tasks are allowed to run on shielded CPUs, if those tasks are not consuming all the available power then there is free CPU time which is lost. The ARTiS scheduler extends this second approach by also allowing normal tasks to be executed on those processors as long as they are not endangering the real-time properties. 2 ARTiS: Asymmetric Real-Time Scheduler Our proposition is a contribution to the definition of a real-time Linux extension that targets SMPs. Furthermore, the programming model we promote is based on a user-space programming of the real-time tasks: the programmer uses the usual POSIX and/or Linux API to define his applications. These tasks are real-time in the sense that they are identified with a high priority and are not perturbed by any non real-time activities. For these tasks, we are targeting a maximum response time below 300μs. This limit was obtained after a study by the industrial partners concerning their requirements. To take advantage of an SMP architecture, an operating system needs to take into account the shared memory facility, the migration and load-balancing between processors, and the communication patterns between tasks. The complexity of such an operating system makes it look more like a general purpose operating system (GPOS) than a dedicated real-time operating system (RTOS). An RTOS on SMP machines must implement all these mechanisms and consider how they interfere with the hard real-time constraints. This may explain why RTOS’s are almost mono-processor dedicated. The Linux kernel is able to efficiently manage SMP platforms, but it is agreed that the Linux kernel has not been designed as an RTOS. Technically, only soft real-time tasks are supported, via the FIFO and round-robin scheduling policies. The ARTiS solution keeps the interests of both GPOS’s and RTOS’s by establishing from the SMP platform an Asymmetric Real-Time Scheduler in Linux. We want to keep the full Linux facilities for each process as well as the SMP Linux properties but we want to improve the real-time behavior too. The core of the ARTiS solution is based on a strong distinction between real-time and non-real-time processors and also on migrating tasks which attempt to disable the preemption on a real-time processor. Partition of the Processors and Processes Processors are partitioned into two sets, an NRT CPU set (Non-Real-Time) and an RT CPU set (Real-Time). Each one has a particular scheduling policy. The purpose is to insure the best interrupt latency for particular processes running in the RT CPU set. Two classes of RT processes are defined. These are standard RT Linux processes, they just differ in their mapping: – Each RT CPU has one or several bound RT Linux tasks, called RT0 (a realtime task of highest priority). Each of these tasks has the guarantee that its RT CPU will stay entirely available to it. Only these user tasks are allowed to become non-preemptible on their corresponding RT CPU. This property insures a latency as low as possible for all RT0 tasks. The RT0 tasks are the hard real-time tasks of ARTiS. Execution of more than one RT0 task on one RT CPU is possible but in this case it is up to the developer to verify the feasibility of such a scheduling. – Each RT CPU can run other RT Linux tasks but only in a preemptible state. Depending on their priority, these tasks are called RT1, RT2... or RT99. To generalize, we call them RT1+. They can use CPU resources efficiently if RT0 tasks do not consume all the CPU time. To keep a low latency for the RT0 tasks, the RT1+ tasks are automatically migrated to an NRT CPU by the ARTiS scheduler when they are about to become non-preemptible (when they call preempt disable() or local irq disable()). The RT1+ tasks are the soft real-time tasks of ARTiS. They have no firm guarantees, but their requirements are taken into account by a best effort policy. They are also the main support of the intensive processing parts of the targeted applications. – The other, non-real-time, tasks are named “Linux tasks” in the ARTiS terminology. They are not related to any real-time requirements. They can coexist with real-time tasks and are eligible for selection by the scheduler as long as the real-time tasks do not require the CPU. As for the RT1+, the Linux tasks will automatically migrate away from an RT CPU if they try to enter into a non-preemptible code section on such a CPU. – The NRT CPUs mainly run Linux tasks. They also run RT1+ tasks which are in a non-preemptible state. To insure the load-balancing of the system, all these tasks can migrate to an RT CPU but only in a preemptible state. When an RT1+ task runs on an NRT CPU, it keeps its high priority above the Linux tasks. ARTiS then supports three different levels of real-time processing: RT0, RT1+ and Linux. RT0 tasks are implemented in order to minimize the jitter due to non-preemptible execution on the same CPU. Note that these tasks are still user-space Linux tasks. RT1+ tasks are soft real-time tasks but they are able to take advantage of the SMP architecture, particularly for intensive computing. Eventually, Linux tasks can run without intrusion on the RT CPUs. Then they can use the full resources of the SMP machines. This architecture is adapted to large applications made of several components requiring different levels of real-time guarantees and of CPU power. Migration Mechanism A particular migration mechanism has been defined. It aims at insuring the low latency of the RT0 tasks. All the RT1+ and Linux tasks running on an RT CPU are automatically migrated toward an NRT CPU when they try to disable the preemption. One of the main requirement is a mechanism without any inter-CPU locks. Such locks are extremely dangerous for the realtime properties if an RT CPU have to wait after an NRT CPU. To effectively migrate the tasks, an NRT CPU and an RT CPU have to communicate via queues. We have implemented a lock-free FIFO with one reader and one writer to avoid any active wait of the ARTiS scheduler based on the algorithm proposed by Valois [9]. Load-Balancing Policy An efficient load-balancing policy allows the full power of the SMP machine to be exploited. Usually a load-balancing mechanism aims to move the running tasks across CPUs in order to insure that no CPU is idle while tasks are waiting to be scheduled. Our case is more complicated because of the introduction of asymmetry and the heavy use of real-time tasks. To minimize the latency on RT CPUs and to provide the best performances for the global system, particular asymmetric load-balancing algorithms have been defined [7]. 3 ARTiS Current Implementation A basic ARTiS API has been defined. It allows the deployment of applications on the current implementation of the ARTiS model, defined as a modification of the 2.6 Linux kernel. A user defines its ARTiS application by configuring the CPUs, identifying the real-time tasks and their processor affinity via a basic /proc interface and some system calls (sched setscheduler()...). The current implementation[5] first consists of a migration mechanism that ensures only preemptible code is executed on an RT CPU. This migration relies on a task FIFO implemented with lock-free access [9]: one writer on the RT CPU and one reader on the NRT CPU. The load-balancer implementation was carried out by improving or specializing several parts of the original Linux one. The main modification was to change from a “pull” policy to a “push” policy: it is the over-loaded CPUs which send tasks to the under-loaded ones. Although it slightly decreases performances because idle CPUs might spend more time idle, this permitted the removal of inter-CPU locks. The load estimation of a processor has also been enhanced in order to correctly estimate a load of a real-time task (which will never share CPU power with other tasks). This enhancement permit better equity among Linux tasks. Additionally, the designation criteria has been modified to favor the presence of RT1+ (resp. Linux) tasks on RT CPUs (resp. NRT CPUs) because that is where latencies are the smallest. Similarly, tasks which are unlikely to block the interrupts soon (according to statistics about their previous behavior) will be preferred for going to an RT CPU. A specific tool was designed to test the load-balancer correctness. It allows to run a specific set of tasks characterized by properties (CPU usage, scheduler priority, processor affinity...) to be launched in a very deterministic way. Some statistics about the run are provided but the interpretation of the results is not straightforward. Additional studies need to be done on the load-balancer implementation. 4 Performance Evaluation While implementing the ARTiS kernel, some experiments were conducted in order to evaluate the potential benefits of the approach in terms of interrupt latency. We distinguished two types of latency, one associated with the kernel and the other one associated with user tasks. Measurement Method The experiment consisted of measuring the elapsed time between the hardware generation of an interrupt and the execution of the code concerning this interrupt. The experimentation protocol was written with the wish to stay as close as possible to the common mechanisms employed by real-time tasks. The measurement task sets up the hardware so it generates the interrupt at a precisely known time, then it gets unscheduled and wait for the interrupt information to occur. Once the information is sent, the task is woken up, the current time is saved and the next measurement starts. For one interrupt there are four associated times, corresponding to different locations in the executed code (figure 1):